Design and Implementation of a Distributed Search and Advertising System

This paper reports on the authors’ experience in developing and running a distributed system for Web search and search-based document placement (advertising). Distributed architecture was chosen to address scalability issues and to provide seamless search service for the ‘invisible Web’. The system consists of topical document collections and request routing infrastructure. Request routing to the appropriate document collection is performed for both search and placement requests. Other features of the system include support for structured attribute-based search, topic-specific (focused) Web crawling, and payment transactions. Security issues arising from the distributed nature of the system as well as from revenue generation are discussed. Important architectural and design decisions are presented and analysed in the light of implementation and pilot operation of the system.